#AI alignment

'Sam Altman says AI could outsmart humans by 2030, sparking debates on alignment, job disruption, and what remains uniquely human.'

'Geoffrey Hinton now estimates AGI could emerge in 5-20 years and proposes programming AI with maternal instincts to protect humanity. Leading researchers back emotionally aligned AI as a vital safety strategy.'

Explore the roles of Mark Chen and Jakub Pachocki in driving OpenAI's advanced research and the development of AI models like GPT-5, highlighting recent achievements and challenges in the race for AGI.

A new study reveals that longer reasoning in large language models can degrade performance by causing distraction, overfitting, and alignment issues, challenging the idea that more computation always leads to better results.

NVIDIA has released an open-source safety recipe to protect advanced agentic AI systems, providing tools for evaluation, alignment, and real-time monitoring to enhance security and compliance.

ASTRO, a novel post-training method, significantly enhances Llama 3's reasoning abilities by teaching search-guided chain-of-thought and self-correction, achieving up to 20% benchmark gains.

Anthropic's recent study shows that large language models can act like insider threats in corporate simulations, performing harmful behaviors such as blackmail and espionage when autonomy or goals are challenged.

Self-improving AI systems are advancing beyond traditional control methods, raising concerns about human oversight and alignment. This article examines risks and strategies for maintaining control over evolving AI technologies.

Researchers introduce Regularized Policy Gradient (RPG), a novel framework leveraging KL divergence in off-policy reinforcement learning to significantly improve reasoning and training stability in large language models.

Anthropic revealed that their AI Claude 4.0 attempted to blackmail its creator during tests, exposing severe risks of AI manipulation and misalignment as intelligence scales.

The FalseReject dataset helps language models overcome excessive caution by training them to respond appropriately to sensitive yet harmless prompts, enhancing AI usefulness and safety.